Allow configuring `cascade` deletion in DABs by ronaldz-db · Pull Request #5846 · databricks/cli

ronaldz-db · 2026-07-08T01:32:43Z

Changes

Add a cascade_on_destory field to the pipeline bundle resource that controls whether destroying a pipeline also cascades to its datasets. When unset, we maintain the existing default of true.

This change applies to the direct engine. We have a separate change on the Terraform side: databricks/terraform-provider-databricks#5860

Why

Previously, pipeline deletion would automatically cascade to its tables. After user feedback, we decided to make this configurable in the DeletePipeline API. This extends support to DABs as well, which was another common user ask.

Tests

Unit tests

Add a `cascade` field to the pipeline bundle resource that controls whether destroying a pipeline also deletes its datasets (materialized views, streaming tables, and views). When unset, the server default applies (cascade); set `cascade: false` to retain the datasets on destroy. The field is delete-time only: it is not part of the pipeline spec, so it is never sent on create/update. The direct engine persists it in state (via a PipelineState wrapper around CreatePipeline, mirroring the sql_warehouse lifecycle pattern) and reads it at delete time, force-sending cascade so an explicit false survives query-string omitempty (honored for query params as of databricks-sdk-go v0.152.0). A cascade-only change is a state-only update with no pipeline API call. cascade is classified input_only so it does not show remote drift. The terraform engine drops the field for now: the pinned provider has no such attribute yet (pending terraform-provider-databricks#5860), so terraform-engine support is a follow-up. Co-authored-by: Isaac

github-actions · 2026-07-08T01:34:25Z

Approval status: pending

`/acceptance/bundle/` - needs approval

29 files changed
Suggested: @pietern
Also eligible: @denik, @janniklasrose, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

`/acceptance/pipelines/` - needs approval

14 files changed
Suggested: @lennartkats-db
Also eligible: @kanterov, @jefferycheng1

`/bundle/` - needs approval

12 files changed
Suggested: @pietern
Also eligible: @denik, @janniklasrose, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/cascade-on-destroy.md
Based on git history:

@pietern -- recent work in bundle/config/mutator/, bundle/schema/, bundle/phases/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.

See OWNERS for ownership rules.}

lennartkats-db · 2026-07-09T11:41:11Z

+  pipelines:
+    my_pipeline:
+      name: test-pipeline-cascade
+      cascade: false


Could you make sure the default template also sets purge_on_destroy: false? With a very short comment to the side explaining it? Could be a separate PR.

ack will put this change in a follow-up PR.

… message fix - Rename the pipeline config field cascade -> cascade_on_destroy, matching the Lakebase purge_on_delete precedent (per review discussion). - Add ValidateCascadeOnDestroy mutator: hard error when the field is used with the terraform engine, which does not support it yet. - Fix the destroy approval message so a pipeline with cascade_on_destroy: false is not described as deleting its STs/MVs, and mention the property otherwise. - Comment why ForceSendFields is needed in DoDelete. - Regenerate schema, refschema, and affected acceptance outputs. Co-authored-by: Isaac

ronaldz-db · 2026-07-10T07:00:42Z

+  pipelines:
+    my_pipeline:
+      name: test-pipeline-cascade
+      cascade: false


ack will put this change in a follow-up PR.

…cade # Conflicts: # NEXT_CHANGELOG.md

Align the pipeline destroy-time attribute name with the DABs config field (databricks/cli#5846), which settled on cascade_on_destroy. Co-authored-by: Isaac

shreyas-goenka · 2026-07-15T22:13:06Z


 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:


Suggested change

Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:

Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain the Streaming Tables and Materialized Views on pipeline deletion:

datasets seems like a vague term to use here? In the context of dashboards, for example, this just means a SQL query.

I could be wrong about this, so if this is standard for pipelines, feel free to keep it as is.

We generally use datasets in pipelines docs now, since pipeline-owned tables have been changing recently. Now, we also allow users to create Views in pipelines. In the near future, we plan to switch to creating regular delta tables as well.

shreyas-goenka · 2026-07-15T22:28:34Z

 type PipelineRemote struct {
 	pipelines.CreatePipeline

+	CascadeOnDestroy *bool `json:"cascade_on_destroy,omitempty"`


Can be omitted from this struct because cascade on destroy will never be set in the remote. It'll never be returned as part of the get response

ah I was confused and thought that there was a strict requirement that PipelineRemote be a superset. But looking closer, I don't think there a benefit to defining for the remote version. Dropped it

shreyas-goenka · 2026-07-15T22:32:41Z

+	deletePipelineWithCascadeMessage = `This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:`
+
+	deletePipelineNoCascadeMessage = `This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines.


Drop the files.watcherExclude block that was unrelated to the cascade_on_destroy feature, per review feedback. Co-authored-by: Isaac

- Add docs URL to the terraform-engine validation error so users can learn about direct deployment mode. - Omit CascadeOnDestroy from PipelineRemote: it is an input-only, delete-time setting the GET response never returns, so drift is suppressed automatically (missing_in_remote). Matches the postgres input-only field pattern. - Propagate errors from pipelineDeletionCascades instead of silently defaulting, and add unit coverage. Co-authored-by: Isaac

github-actions · 2026-07-16T04:38:21Z

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 5846
Commit SHA: 423ca32b692ba3efd28bc2b884659db8add2f764

Checks will be approved automatically on success.

shreyas-goenka · 2026-07-16T07:03:40Z

+
+	v, err := dyn.GetByPath(b.Config.Value(), path)
+	if err != nil {
+		return true, nil


This embeds a client side default here. Treating unset as true. Won't this get out of date once the server side default changes?

I guess there's no good way to handle this scenario? It does mean we'll eventually end up showing wrong messages to people.

Can you add a to-do message here to change the value once the server side default flips?

Yep, added a note here. Unfortunately, there may be a brief period where the DABs default and server-side default are different. But the DABs documentation describes the actual encoded behavior, so we won't be showing the wrong messages

shreyas-goenka · 2026-07-16T07:34:28Z

+
+	v, err := dyn.GetByPath(b.Config.Value(), path)
+	if err != nil {
+		return true, nil


Can you add a to-do message here to change the value once the server side default flips?

shreyas-goenka · 2026-07-16T12:39:29Z

+	deletePipelineWithCascadeMessage = `This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:`
+
+	deletePipelineNoCascadeMessage = `This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines.


GPT 5.6 flagged a crucial bug. @ronaldz-db we need to read the cascade_on_delete value from state when showing the prompt. Otherwise we risk deleting a user's table when tell them that we'll not:

Critical — approval can falsely promise dataset retention. /tmp/isaac-review-cli/bundle/phases/plan.go:46 reads the current YAML, while deletion uses persisted state. If a user changes cascade_on_destroy to false and immediately destroys without deploying, the prompt says datasets are retained, but the API uses the old state and cascades. Derive both behavior and messaging from the same state; add regression coverage for this sequence. Focused package tests pass. Current CI failures are unrelated JFrog OIDC setup failures. REQUEST CHANGES

Can you check if this feedback is valid? I suspect it is because from I recall we indeed only rely on the state for delete / destroy operations.

Please also add an acceptance test to ensure that we do the right thing when a user sets this field but then immediately destroys the bundle the bundle without a deployment.

oh shoot, thanks for the catch. This feedback is valid. I've updated the message to be generated by reading the state value for the direct engine and added a test.

lennartkats-db reviewed Jul 8, 2026

View reviewed changes

Comment thread bundle/direct/dresources/pipeline.go

lennartkats-db reviewed Jul 8, 2026

View reviewed changes

Comment thread acceptance/pipelines/destroy/destroy-pipeline-cascade/databricks.yml Outdated

Comment thread acceptance/pipelines/destroy/destroy-pipeline-cascade/output.txt

andrewnester mentioned this pull request Jul 9, 2026

test-only: Add cascade field to pipeline resource for destroy #5874

Open

lennartkats-db requested changes Jul 9, 2026

View reviewed changes

ronaldz-db mentioned this pull request Jul 9, 2026

Allow configuring cascade deletion for pipelines databricks/terraform-provider-databricks#5860

Open

6 tasks

ronaldz-db added 2 commits July 9, 2026 16:33

clean up

9a2436c

ronaldz-db commented Jul 10, 2026

View reviewed changes

ronaldz-db requested a review from lennartkats-db July 10, 2026 07:12

janniklasrose linked an issue Jul 11, 2026 that may be closed by this pull request

Enable cascade delete feature on pipeline for databricks deploy #5841

Open

Merge remote-tracking branch 'origin/main' into pipelines-destroy-cas…

5021a52

…cade # Conflicts: # NEXT_CHANGELOG.md

shreyas-goenka reviewed Jul 15, 2026

View reviewed changes

ronaldz-db added 2 commits July 16, 2026 00:01

Revert unrelated .vscode/settings.json watcherExclude change

cb61ad7

Drop the files.watcherExclude block that was unrelated to the cascade_on_destroy feature, per review feedback. Co-authored-by: Isaac

ronaldz-db requested a review from shreyas-goenka July 16, 2026 04:44

shreyas-goenka reviewed Jul 16, 2026

View reviewed changes

shreyas-goenka approved these changes Jul 16, 2026

View reviewed changes

lennartkats-db approved these changes Jul 16, 2026

View reviewed changes

shreyas-goenka requested changes Jul 16, 2026

View reviewed changes

ronaldz-db requested a review from shreyas-goenka July 16, 2026 23:54

	Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
	Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain the Streaming Tables and Materialized Views on pipeline deletion:

Uh oh!

Conversation

ronaldz-db commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/acceptance/bundle/ - needs approval

/acceptance/pipelines/ - needs approval

/bundle/ - needs approval

General files (require maintainer)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ronaldz-db commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/acceptance/pipelines/` - needs approval

`/bundle/` - needs approval

shreyas-goenka Jul 16, 2026 •

edited

Loading

shreyas-goenka Jul 16, 2026 •

edited

Loading

shreyas-goenka Jul 16, 2026 •

edited

Loading